Document plugin registration and expand plugin tests #3
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Scenario,Metric,Client,Tokenizer(and otherObjectSpec-backed components) can be registered and instantiated.helm-run.Description
docs/importing_custom_modules.mdto enumerate plugin categories (run specs vs ObjectSpec-backed classes), document discovery/registration paths, add a compact example showing@run_spec_function(...)usage andObjectSpec-referenced classes, and clarifyPYTHONPATHguidance.test_import_user_plugins_supports_object_spec_pluginstosrc/helm/benchmark/test_plugins.pywhich creates a temporary module that defines a run spec andCustomScenario,CustomMetric,CustomClient, andCustomTokenizer, then exercisesimport_user_plugins,create_scenario,create_metric,create_object(ClientSpec),create_object(TokenizerSpec)andget_run_spec_function.ObjectSpec+create_object/create_metric/create_scenario) are exercised by the new test.Testing
test_import_user_plugins_supports_object_spec_pluginswas added tosrc/helm/benchmark/test_plugins.pybut has not been run here.src/helm/benchmark/test_plugins.pywere left intact and the new test complements their coverage by targetingObjectSpec-backed components.Codex Task